2D pathfinder visualization

ABSTRACT

In one embodiment, a medical apparatus, includes a medical instrument configured to move within a passage in a body of a patient, a position tracking system to track coordinates of the medical instrument within the body, a display screen, and a processor to register the position tracking system and a 3D computerized tomography (CT) image of at least a part of the body within a common frame of reference, find a 3D path of the medical instrument through the passage from a start point to a termination point within the common frame of reference, compute a direction to the 3D path from the medical instrument responsively to the tracked coordinates, and render and simultaneously display on the display screen respective 2D CT slices, based on the 3D CT image, including respective 2D indications of the direction to the 3D path from the instrument projected onto the respective 2D CT slices.

FIELD OF THE INVENTION

The present invention relates to medical systems, and in particular, butnot exclusively, to path visualization.

BACKGROUND

In image-guided surgery (IGS), a medical practitioner uses instrumentsthat are tracked in real time within the body, so that positions and/ororientations of the instruments can be presented on images of apatient's anatomy during the surgical procedure. In many IGS scenariosan image of the patient is prepared in one modality, such as magneticresonance imaging (MRI) or computerized tomography (CT), and theinstrument tracking uses a different modality, such as electromagnetictracking. In order for the tracking to be effective, frames of referenceof the two modalities are registered with each other.

US Patent Publication 2011/0236868 of Bronstein, et al., issued as U.S.Pat. No. 10,580,325 on Mar. 3, 2020, describes a method of performingcomputerized simulations of image-guided procedures. The method maycomprise receiving medical image data of a specific patient. Apatient-specific digital image-based model of an anatomical structure ofthe specific patient may be generated based on the medical image data. Acomputerized simulation of an image-guided procedure may be performedusing the digital image-based model. Medical image data, the image-basedmodel and a simulated medical tool model may be simultaneouslydisplayed.

US Patent Publication 2017/0151027 of Walker, et al., issued as U.S.Pat. No. 10,143,529 on Dec. 4, 2018, describes systems and methods fordriving a flexible medical instrument to a target in an anatomical spacewith robotic assistance. The flexible instrument may have a trackingsensor embedded therein. An associated robotic control system may beprovided, which is configured to register the flexible instrument to ananatomical image using data from the tracking sensor and identify one ormore movements suitable for navigating the instrument towards anidentified target. In some embodiments, the robotic control systemdrives or assists in driving the flexible instrument to the target.

US Patent Publication 2016/0174874 of Averbuch, et al., issued as U.S.Pat. No. 10,285,623 on May 14, 2019, describes a registration methodwhereby a sensor-based approach is used to establish initialregistration and whereby upon the commencement of navigating anendoscope, image-based registration methods are used in order to moreaccurately maintain the registration between the endoscope location andpreviously-acquired images. A six-degree-of-freedom location sensor isplaced on the probe in order to reduce the number of previously-acquiredimages that must be compared to a real-time image obtained from theendoscope.

US Patent Publication 2005/0228250 of Bitter, et al., now abandoned,describes a user interface including an image area that is divided intoa plurality of views for viewing corresponding 2-dimensional and3-dimensional images of an anatomical region. Tool control panes can besimultaneously opened and accessible. The segmentation pane enablesautomatic segmentation of components of a displayed image within auser-specified intensity range or based on a predetermined intensity.

US Patent Publication 2007/0276214 of Dachille, et al., now abandoned,describes an imaging system for automated segmentation and visualizationof medical images and includes an image processing module forautomatically processing image data using a set of directives toidentify a target object in the image data and process the image dataaccording to a specified protocol, a rendering module for automaticallygenerating one or more images of the target object based on one or moreof the directives and a digital archive for storing the one or moregenerated images. The image data may be DICOM-formatted image data),wherein the imaging processing module extracts and processes meta-datain DICOM fields of the image data to identify the target object. Theimage processing module directs a segmentation module to segment thetarget object using processing parameters specified by one or more ofthe directives.

U.S. Pat. No. 5,371,778 to Yanof, et al., describes a CT scanner thatnon-invasively examines a volumetric region of a subject and generatesvolumetric image data indicative thereof. An object memory stores thedata values corresponding to each voxel of the volume region. An affinetransform algorithm operates on the visible faces of the volumetricregion to translate the faces from object space to projections of thefaces onto a viewing plane in image space. An operator control consoleincludes operator controls for selecting an angular orientation of aprojection image of the volumetric region relative to a viewing plane,i.e. a plane of the video display. A cursor positioning trackball inputsi- and j-coordinate locations in image space which are converted into acursor crosshair display on the projection image. A depth dimension kbetween the viewing plane and the volumetric region in a viewingdirection perpendicular to the viewing plane is determined. The (i,j,k)image space location of the cursor is operated upon by the reverse ofthe selected transform to identify a corresponding (x,y,z) cursorcoordinate in object space. The cursor coordinate in object space istranslated into corresponding addresses of the object memory fortransverse, coronal, and sagittal planes through the volumetric region.

U.S. Pat. No. 10,188,465 to Gliner, et al., describes a method includingreceiving a computerized tomography scan of at least a part of a body ofa patient, and identifying voxels of the scan that correspond to regionsin the body that are traversable by a probe inserted therein. The methodalso includes displaying the scan on a screen and marking thereonselected start and termination points for the probe. A processor finds apath from the start point to the termination point consisting of aconnected set of the identified voxels. The processor also uses the scanto generate a representation of an external surface of the body anddisplays the representation on the screen. The processor then renders anarea of the external surface surrounding the path locally transparent inthe displayed representation, so as to make visible on the screen aninternal structure of the body in a vicinity of the path.

US Patent Publication 2018/0303550 of Altmann, et al., issued as U.S.Pat. No. 11,026,747 on Jun. 8, 2021, describes a method forvisualization includes registering, within a common frame of reference,a position tracking system and a three-dimensional (3D) computerizedtomography (CT) image of at least a part of a body of a patient. Alocation and orientation of at least one virtual camera are specifiedwithin the common frame of reference. Coordinates of a medical toolmoving within a passage in the body are tracked using the positiontracking system. A virtual endoscopic image, based on the 3D CT image,of the passage in the body is rendered and displayed from the specifiedlocation and orientation, including an animated representation of themedical tool positioned in the virtual endoscopic image in accordancewith the tracked coordinates.

SUMMARY

There is provided in accordance with an embodiment of the presentdisclosure, a medical apparatus, including a medical instrument, whichis configured to move within a passage in a body of a patient, aposition tracking system, which is configured to track coordinates ofthe medical instrument within the body, a display screen, and aprocessor, which is configured to register the position tracking systemand a three-dimensional (3D) computerized tomography (CT) image of atleast a part of the body within a common frame of reference, find a 3Dpath of the medical instrument through the passage from a given startpoint to a given termination point within the common frame of reference,compute a direction to the 3D path from the medical instrumentresponsively to the tracked coordinates, and render and simultaneouslydisplay on the display screen respective two-dimensional (2D) CT slices,based on the 3D CT image, including respective 2D indications of thedirection to the 3D path from the medical instrument projected onto therespective 2D CT slices.

Further in accordance with an embodiment of the present disclosure therespective 2D indications of the direction to the 3D path from themedical instrument include respective arrows.

Still further in accordance with an embodiment of the present disclosurethe processor is configured to render and simultaneously display on thedisplay screen the respective two-dimensional (2D) CT slices, based onthe 3D CT image, including respective 2D indications of the direction tothe 3D path from the medical instrument projected into the respective 2DCT slices, and a representation of the medical instrument responsivelyto the tracked coordinates.

Additionally, in accordance with an embodiment of the present disclosurethe processor is configured to compute a direction to a closest point ofthe 3D path from the medical instrument responsively to the trackedcoordinates.

Moreover, in accordance with an embodiment of the present disclosure theprocessor is configured to compute a 3D vector to the 3D path from themedical instrument responsively to the tracked coordinates.

Further in accordance with an embodiment of the present disclosure theprocessor is configured to render and simultaneously display on thedisplay screen the respective 2D CT slices, based on the 3D CT image,including the respective 2D indications of the 3D vector to the 3D pathfrom the medical instrument projected onto the respective 2D CT slices.

Still further in accordance with an embodiment of the present disclosurethe processor is configured to render and simultaneously display on thedisplay screen three respective two-dimensional (2D) CT slices, based onthe 3D CT image, including three respective 2D indications of thedirection to the 3D path from the medical instrument projected onto thethree respective 2D CT slices.

Additionally, in accordance with an embodiment of the present disclosurethe three respective 2D slices respectively include a Coronal view, aSagittal view, and a Transversal view.

Moreover in accordance with an embodiment of the present disclosure theprocessor is configured to compute a distance to the 3D path from themedical instrument responsively to the tracked coordinates, and renderand simultaneously display on the display screen the respectivetwo-dimensional (2D) CT slices, based on the 3D CT image, includingrespective 2D indications of the direction and the distance to the 3Dpath from the medical instrument projected onto the respective 2D CTslices.

Further in accordance with an embodiment of the present disclosure theprocessor is configured to render and simultaneously display on thedisplay screen the respective two-dimensional (2D) CT slices, based onthe 3D CT image, including the respective 2D indications of thedirection to the 3D path from the medical instrument projected onto therespective 2D CT slices, and a virtual endoscopic image, based on the 3DCT image, of the passage in the body viewed from at least one respectivelocation of at least one respective virtual camera including an animatedrepresentation of the medical instrument positioned in the virtualendoscopic image in accordance with the tracked coordinates, and a 3Dindication of the direction to the 3D path from the medical instrument.

Still further in accordance with an embodiment of the present disclosurethe position tracking system includes an electromagnetic trackingsystem, which includes one or more magnetic field generators positionedaround the part of the body and a magnetic field sensor at a distal endof the medical instrument.

There is also provided in accordance with another embodiment of thepresent disclosure, a medical method, including tracking coordinates ofa medical instrument within a body of a patient using a positiontracking system, the medical instrument being configured to move withina passage in the body of the patient, registering the position trackingsystem and a three-dimensional (3D) computerized tomography (CT) imageof at least a part of the body within a common frame of reference,finding a 3D path of the medical instrument through the passage from agiven start point to a given termination point within the common frameof reference, computing a direction to the 3D path from the medicalinstrument responsively to the tracked coordinates, and rendering andsimultaneously displaying on a display screen respective two-dimensional(2D) CT slices, based on the 3D CT image, including respective 2Dindications of the direction to the 3D path from the medical instrumentprojected onto the respective 2D CT slices.

Additionally, in accordance with an embodiment of the present disclosurethe respective 2D indications of the direction to the 3D path from themedical instrument include respective arrows.

Moreover, in accordance with an embodiment of the present disclosure therendering and simultaneously displaying includes rendering andsimultaneously displaying on the display screen the respectivetwo-dimensional (2D) CT slices, based on the 3D CT image, includingrespective 2D indications of the direction to the 3D path from themedical instrument projected into the respective 2D CT slices, and arepresentation of the medical instrument responsively to the trackedcoordinates.

Further in accordance with an embodiment of the present disclosure thecomputing includes computing a direction to a closest point of the 3Dpath from the medical instrument responsively to the trackedcoordinates.

Still further in accordance with an embodiment of the present disclosurethe computing includes computing a 3D vector to the 3D path from themedical instrument responsively to the tracked coordinates.

Additionally, in accordance with an embodiment of the present disclosurethe rendering and simultaneously displaying includes rendering andsimultaneously displaying on the display screen the respective 2D CTslices, based on the 3D CT image, including the respective 2Dindications of the 3D vector to the 3D path from the medical instrumentprojected onto the respective 2D CT slices.

Moreover, in accordance with an embodiment of the present disclosure therendering and simultaneously displaying includes rendering andsimultaneously displaying on the display screen three respectivetwo-dimensional (2D) CT slices, based on the 3D CT image, includingthree respective 2D indications of the direction to the 3D path from themedical instrument projected onto the three respective 2D CT slices.

Further in accordance with an embodiment of the present disclosure thethree respective 2D slices respectively include a Coronal view, aSagittal view, and a Transversal view.

Still further in accordance with an embodiment of the presentdisclosure, the method includes computing a distance to the 3D path fromthe medical instrument responsively to the tracked coordinates, andwherein the rendering and simultaneously displaying includes renderingand simultaneously displaying on the display screen the respectivetwo-dimensional (2D) CT slices, based on the 3D CT image, includingrespective 2D indications of the direction and the distance to the 3Dpath from the medical instrument projected onto the respective 2D CTslices.

Additionally, in accordance with an embodiment of the present disclosurethe rendering and simultaneously displaying includes rendering andsimultaneously displaying on the display screen the respectivetwo-dimensional (2D) CT slices, based on the 3D CT image, including therespective 2D indications of the direction to the 3D path from themedical instrument projected onto the respective 2D CT slices, and avirtual endoscopic image, based on the 3D CT image, of the passage inthe body viewed from at least one respective location of at least onerespective virtual camera including an animated representation of themedical instrument positioned in the virtual endoscopic image inaccordance with the tracked coordinates, and a 3D indication of thedirection to the 3D path from the medical instrument.

There is also provided in accordance with still another embodiment ofthe present disclosure a software product, including a non-transientcomputer-readable medium in which program instructions are stored, whichinstructions, when read by a central processing unit (CPU), cause theCPU to track coordinates of a medical instrument within a body of apatient using a position tracking system, the medical instrument beingconfigured to move within a passage in the body of the patient, registerthe position tracking system and a three-dimensional (3D) computerizedtomography (CT) image of at least a part of the body within a commonframe of reference, find a 3D path of the medical instrument through thepassage from a given start point to a given termination point within thecommon frame of reference, compute a direction to the 3D path from themedical instrument responsively to the tracked coordinates, and renderand simultaneously display on a display screen respectivetwo-dimensional (2D) CT slices, based on the 3D CT image, includingrespective 2D indications of the direction to the 3D path from themedical instrument projected onto the respective 2D CT slices.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood from the following detaileddescription, taken in conjunction with the drawings in which:

FIG. 1 is a partly pictorial, partly block diagram view of a medicalsystem constructed and operative in accordance with an embodiment of thepresent invention;

FIG. 2 is a flowchart including steps in a method of three-dimensionalpath visualization for use in the apparatus of FIG. 1 in accordance withan embodiment of the present invention;

FIG. 3 is a flowchart including steps of a method of finding a paththrough a passage using the apparatus of FIG. 1;

FIGS. 4-9 are schematic diagrams illustrating the steps of the method ifthe flowchart of FIG. 3;

FIG. 10 is a flowchart including steps in a method of computing segmentsof a path and computing locations of virtual cameras along the path foruse in the apparatus of FIG. 1;

FIGS. 11 and 12 are schematic diagrams illustrating the steps of themethod of the flowchart of FIG. 10;

FIG. 13 is a flowchart including steps in a method of selecting a camerafor use in the apparatus of FIG. 1;

FIGS. 14 and 15 are schematic diagrams illustrating the steps of themethod of the flowchart of FIG. 13;

FIG. 16 is a flowchart including steps in a method of computing anorientation and shifting the location of a virtual camera for use in theapparatus of FIG. 1;

FIGS. 17 and 18 are schematic diagrams illustrating the steps of themethod of the flowchart of FIG. 16;

FIG. 19 is a flowchart including steps in a method of rendering atransition for use in the apparatus of FIG. 1;

FIG. 20 is a schematic diagram illustrating the steps of the method ofthe flowchart of FIG. 19;

FIGS. 21-23 are schematic virtual endoscopic images rendered anddisplayed by the apparatus of FIG. 1;

FIG. 24 is a flowchart including steps in a method of two-dimensionalpath visualization for use in the apparatus of FIG. 1 in accordance withan embodiment of the present invention; and

FIG. 25 is a schematic view of a combined 2D and 3D path visualizationillustrating the steps of the method of the flowchart of FIG. 24.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

During medical procedures within the nasal passages, such as sinuplastyoperations, it is impossible to directly visualize what is happeningwithout insertion of an endoscope into the sinuses. Insertion of anendoscope is problematic, however, because of the tight spaces involved,as well as the extra cost of the endoscope. Furthermore, endoscopes foruse in the nasal passages are typically rigid instruments, which cannotmake turns or provide views from sinus cavities back toward the sinusopening.

Embodiments of the present invention that are described herein addressthis problem by generating virtual endoscopic views of the procedurefrom virtual cameras, similar to what would be seen by an actualendoscope positioned at the locations of the virtual cameras within thenasal passages. The virtual endoscopic views show the anatomy as well asthe medical instrument moving through the anatomy. As the medicalinstrument moves along a passage, the virtual cameras used to generatethe virtual endoscopic views passes from one virtual camera to the otherresponsively to tracked coordinates of the medical instrument.

These virtual endoscope views may be used, for example, in visualizingthe location and orientation of a guidewire relative to the anatomy, aswell as other instruments, such as a suction tool or a shaving tool(debrider).

Moving from one virtual camera to the other provides a more stable viewof the anatomy than placing a virtual camera on the distal tip of themedical instrument whereby the virtual camera is always “moving” as thedistal tip moves leading to a jumpy or choppy video of the anatomy whichis very difficult to follow.

Furthermore, although the embodiments disclosed hereinbelow are directedspecifically to visualization within the nasal passages, the principlesof the present invention may similarly be applied within other spaces inthe body, particularly in narrow passages in which actual opticalendoscopy is unavailable or difficult to use.

Prior to the medical procedure, a CT image of the patient's head,including the sinuses, is acquired, and a position tracking system, suchas an electromagnetic tracking system, is registered with the CT image.A position sensor is attached to the distal end of the guidewire orother instrument, and the distal end is thus tracked, in location andorientation, relative to the registered CT image, as it is inserted intothe sinuses. The CT image of the head is processed in order to generateand display images of the 3D volume of the nasal passages.

Inside this 3D volume, an operator of the imaging system, such as asurgeon performing a sinuplasty procedure, can select start andtermination points of a 3D path along which to navigate the medicalinstrument. A suitable 3D path from the start to the termination pointis computed, for example, using a path finding algorithm and data fromthe CT image indicating which voxels of the CT image include materialsuitable for traversing, such as air or liquid.

The computed 3D path is automatically divided into segments with turningpoints between the segments above a threshold turning value. The virtualcameras are positioned around these turning points. Additional virtualcameras may be automatically positioned between the turning points ifthere is no line of sight between the virtual cameras positioned at theturning points and/or if the distance between the turning points exceedsa given value.

The orientation of the optical axis of each virtual camera is computed.The orientation may be computed using any suitable method. In someembodiments, the orientation may be computed based on an averagedirection of vectors from a virtual camera to locations along the pathuntil the next virtual camera. In other embodiments, the orientation maybe computed as a direction parallel to the path at the location of therespective virtual camera. The field of view of the virtual cameras maybe fixed, e.g., to 90 degrees or any suitable value, or set according tothe outer limits of the relevant segment of the path served by therespective virtual cameras with an additional tolerance to allow fordeviations from the path.

In some embodiments, the location of a virtual camera may be shiftedbackwards in an opposite direction to the computed average direction.The virtual camera may be shifted back by any suitable distance, forexample, until the camera is shifted back to solid material such astissue or bone. Shifting the virtual cameras back may result in a betterview of the medical instrument within the respective virtual endoscopicimages particularly when the medical instrument is very close to therespective virtual cameras, and may result in a better view of thesurrounding anatomy.

As the medical tool moves through the passages, the respective virtualcameras are selected for rendering and displaying respective virtualendoscopic images according to a camera selection method. In someembodiments, the camera selection method includes finding a closestcamera to the tracked coordinates of the medical instrument and thenfinding which side of a bisector (plane) associated with the closestcamera the tracked coordinates fall. If the tracked coordinates fall onthe side of the bisector further down the computed path (in thedirection of travel of the medical instrument) from the closest camera,the closest camera is selected for rendering. If the tracked coordinatesfall on the side of the bisector closest to the current virtual camera,the current virtual camera continues to provide its endoscopic image.The bisector associated with the closest camera may be defined as aplane perpendicular to the computed path at the point of the closestcamera. In other embodiments, the passages may be divided into regionsbased on the segments with the virtual cameras being selected accordingto the region in which the tracked coordinates are disposed.

The transition between two virtual cameras and therefore the transitionbetween the associated virtual endoscopic images may be a smoothtransition or a sharp transition. In some embodiments, a smoothtransition between the two respective virtual cameras may be performedby finding locations of addition virtual cameras on the path between thetwo the virtual cameras and then successively rendering respectivetransitional virtual endoscope images viewed from the locations of theadditional virtual cameras.

Although 3D images may be used to help a physician navigate a medicalinstrument along a computed path, some physicians are not comfortablewith a 3D image presentation for this purpose.

Embodiments of the present invention solve the above problem byrendering and displaying, on a display, two-dimensional (2D) CT slicesof a CT image, such as 2D Coronal, Sagittal, and Transversal slices of a3D CT image showing on each 2D image a direction in which to move themedical instrument in order to return the medical instrument to thecomputed path. In some embodiments, a 3D image, such as a virtualendoscopic image, showing a representation of the medical tool may bedisplayed alongside the 2D CT slices.

In some embodiments, a processor computes a direction, and optionally adistance, and/or a 3D vector, to the 3D path (e.g., to a closest pointon the 3D path) from the medical instrument responsively to the trackedcoordinates of the medical instrument. The processor renders andsimultaneously displays on the display screen respective 2D CT slices,based on a 3D CT image, including respective 2D indications (e.g.,arrows or other symbols) of the direction, and optionally the distance,and/or the 3D vector, to the 3D path from the medical instrumentprojected onto the respective 2D CT slices, and optionally arepresentation of the medical instrument responsively to the trackedcoordinates. In some embodiments, the respective 2D slices respectivelyinclude a Coronal view, a Sagittal view, and a Transversal view.

System Description

Reference is now made to FIG. 1, which is a partly pictorial, partlyblock diagram view of a medical apparatus 20 constructed and operativein accordance with an embodiment of the present invention. In thefollowing description a medical instrument 21 of apparatus 20 is assumedto be used to perform a medical procedure on a patient 22. The medicalinstrument 21 is configured to move within a passage in a body of thepatient 22.

The medical apparatus 20 includes a position tracking system 23, whichis configured to track coordinates of the medical instrument 21 withinthe body. In some embodiments, the position tracking system 23 comprisesan electromagnetic tracking system 25, which comprises one or moremagnetic field generators 26 positioned around the part of the body andone or more magnetic field sensors 32 at a distal end of the medicalinstrument 21. In one embodiment, the magnetic field sensors 32comprises a single axis coil and a dual axis coil which act as magneticfield sensors and which are tracked during the procedure by theelectromagnetic tracking system 25. For the tracking to be effective, inapparatus 20 frames of reference of a CT (computerized tomography) imageof patient 22 and of the electromagnetic tracking system 25 areregistered described in more detail with reference to FIGS. 2 and 3.While the CT image may typically comprise a magnetic resonance imaging(MRI) image or a fluoroscopic image, in the description herein the imageis assumed to comprise, by way of example, a fluoroscopic CT image. Insome embodiments, the position tracking system 23 may track coordinatesof the medical instrument 21 using any suitable tracking method, forexample, based on current or impedance distributions over body surfaceelectrodes, or based on ultrasound transducers.

Prior to and during the sinus procedure, a magnetic radiator assembly24, comprised in the electromagnetic tracking system 25, is positionedbeneath the patient's head. The magnetic radiator assembly 24 comprisesthe magnetic field generators 26 which are fixed in position and whichtransmit alternating magnetic fields into a region 30 wherein the headof patient 22 is located. Potentials generated by the single axis coilof the magnetic field sensor(s) 32 in region 30, in response to themagnetic fields, enable its position and its orientation to be measuredin the magnetic tracking system's frame of reference. The position canbe measured in three linear dimensions (3D), and the orientation can bemeasured for two axes that are orthogonal to the axis of symmetry of thesingle axis coil. However, the orientation of the single axis coil withrespect to its axis of symmetry cannot be determined from the potentialsgenerated by the coil.

The same is true for each of the two coils of the dual axis coil of themagnetic field sensor 32. That is, for each coil the position in 3D canbe measured, as can the orientation with respect to two axes that areorthogonal to the coil axis of symmetry, but the orientation of the coilwith respect to its axis of symmetry cannot be determined.

By way of example, radiators 26 of assembly 24 are arranged in anapproximately horseshoe shape around the head of patient 22. However,alternate configurations for the radiators of assembly 24 will beapparent to those having skill in the art, and all such configurationsare assumed to be comprised within the scope of the present invention.

Prior to the procedure, the registration of the frames of reference ofthe magnetic tracking system with the CT image may be performed bypositioning a magnetic sensor at known positions, such as the tip of thepatient's nose, of the image. However, any other convenient system forregistration of the frames of reference may be used as described in moredetail with reference to FIG. 2.

Elements of apparatus 20, including radiators 26 and magnetic fieldsensor 32, are under overall control of a system processor 40. Processor40 may be mounted in a console 50, which comprises operating controls 58that typically include a keypad and/or a pointing device such as a mouseor trackball. Console 50 connects to the radiators and to the magneticfield sensor 32 via one or more cables 60 and/or wirelessly. A physician54 uses operating controls 58 to interact with the processor 40 whileperforming the medical procedure using apparatus 20. While performingthe procedure, the processor may present results of the procedure on adisplay screen 56.

Processor 40 uses software stored in a memory 42 to operate apparatus20. The software may be downloaded to processor 40 in electronic form,over a network, for example, or it may, alternatively or additionally,be provided and/or stored on non-transitory tangible media, such asmagnetic, optical, or electronic memory.

Reference is now made to FIG. 2, which is a flowchart 70 including stepsin a method of three-dimensional path visualization for use in theapparatus 20 of FIG. 1 in accordance with an embodiment of the presentinvention.

The position tracking system 23 (FIG. 1) is configured to track (block72) coordinates of the distal end of the medical instrument 21 (FIG. 1)within the body. The processor 40 (FIG. 1) is configured to register(block 74) the position tracking system 23 and a three-dimensional (3D)computerized tomography (CT) image of at least a part of the body withina common frame of reference. The registration may be performed by anysuitable registration technique. For example, but not limited to, aregistration method described in U.S. Patent Publication 2017/0020411,issued as U.S. Pat. No. 10,638,954 on May 5, 2020, or 2019/0046272. Asdescribed in the latter patent publication, for example, processor 40may analyze the CT image to identify respective locations of thepatient's eyes in the image, thus defining a line segment joining theserespective locations. In addition, processor 40 identifies a voxelsubset in the CT that overlies bony sections of the head, along a secondline segment parallel to the first line segment and a third line segmentperpendicular to the first line segment. The physician 54 positions aprobe in proximity to the bony sections and thus measures positions onthe surface of the head overlying the bony sections. Processor 40computes the correspondence between these measured positions and thevoxel subset in the CT image and thus registers the magnetic trackingsystem 25 with the CT image.

The processor 40 is configured to find (block 76) a 3D path of themedical instrument 21 through a passage from a given start point to agiven termination point. The step of block 76 is described in moredetail with reference to the path finding method of FIGS. 3-9.

The processor 40 is configured to compute (block 78) segments of thecomputed 3D path. The processor 40 is configured to compute (block 80)respective different locations along the 3D path of respective virtualcameras responsively to the computed segments. The steps of blocks 78and 80 are described in more detail with reference to FIGS. 10-12.

The processor 40 is configured to select (block 82) the respectivevirtual cameras for rendering respective virtual endoscopic imagesresponsively to the tracked coordinates of the medical instrument 21 andthe respective locations of the respective virtual cameras within thecommon frame of reference. As the medical instrument 21 moves along the3D path (which may be some distance either side of the path as themedical instrument 21 is not locked to the path), the virtual cameraproviding the virtual endoscopic image is selected according to thetracked coordinates of the medical instrument 21 and control is passedsuccessively from one virtual camera to another as the medicalinstrument 21 moves along the path. The step of block 82 may be repeatedintermittently, for example, each time new tracked coordinates arereceived, for example, in a range between 10 and 100 milliseconds, suchas 50 milliseconds. The step of block 82 is described in more detailwith reference to FIGS. 13-15.

The processor 40 is configured to compute (block 84) respectiveorientations of the respective virtual cameras. The orientation of thecameras is generally a 3D orientation and is defined with respect to arespective optical axis of the respective virtual cameras. In otherwords, the orientation of the cameras is a measure of which directionsthe cameras are facing for optical purposes. The location of the virtualcameras may also be shifted back as described in more detail withreference to FIGS. 16 and 18. The orientations and/or the shifting backmay be computed before a camera is selected, as part of the process ofselecting a camera, or when the medical instrument 21 leaves the fieldof view of the virtual camera currently providing the virtual endoscopicimage.

The processor 40 is configured to render and display (block 86) on thedisplay screen 56 (FIG. 1) the respective virtual endoscopic images,based on the 3D CT image, of the passage in the body viewed from therespective locations and orientations of the respective virtual camerasincluding an animated representation of the medical instrument 21positioned in the respective virtual endoscopic images in accordancewith the tracked coordinates. In other words, based on the location andorientation of the medical instrument 21 and the pre-acquired CT data,the processor 40 renders respective images as they would be captured bythe respective virtual cameras and presents the images on the displayscreen 56, at the step of block 86. The image rendered by any givenselected virtual camera is a projection of a portion of the 3D volumethat would be visible from the camera location onto the virtual imageplane of the camera. The step of block 86 is described in more detailwith reference to FIGS. 21-23.

Reference is now made to FIG. 3, which is a flowchart 90 including stepsof a method of finding a path through a passage using the apparatus 20of FIG. 1. FIGS. 4-9 are schematic diagrams illustrating the steps ofthe method of the flowchart of FIG. 3. The pre-planning componentdescribed with reference to the flowchart 90 is typically implementedprior to performance of the invasive surgery procedure on patient 22(FIG. 1), and determines an optimal path to be followed by invasivemedical instrument 21 (FIG. 1) in the procedure. The pre-planning isassumed to be performed by physician 54 (FIG. 1).

In an initial step (block 100 of the flowchart 90), a computerizedtomography (CT) X-ray scan of the nasal sinuses of patient 22 isperformed, and the data from the scan is acquired by processor 40. As isknown in the art, the scan comprises two-dimensional X-ray “slices” ofthe patient 22, and the combination of the slices generatesthree-dimensional voxels, each voxel having a Hounsfield unit, a measureof radiodensity, determined by the CT scan.

In an image generation step (block 102), physician 54 (FIG. 1) displaysresults of the scan on display screen 56 (FIG. 1). As is known in theart, the results may be displayed as a series of two-dimensional (2D)slices, typically along planes parallel to the sagittal, coronal, and/ortransverse planes of patient 22, although other planes are possible. Thedirection of the planes may be selected by the physician 54.

The displayed results are typically gray scale images, and an example isprovided in FIG. 4, which is a slice parallel to the coronal plane ofpatient 22. The values of the gray scales, from black to white, may becorrelated with the Hounsfield unit (HU) of the corresponding voxels, sothat, as applies to the image of FIG. 4, air having HU=−1000 may beassigned to be black, and dense bone having HU=3000 may be assigned tobe white.

As is known in the art, apart from the values for air and water, whichby definition are respectively −1000 and 0, the value of the Hounsfieldunit of any other substance or species, such as dense bone, isdependent, inter alia, on the spectrum of the irradiating X-rays used toproduce the CT scans referred to herein. In turn, the spectrum of theX-rays depends on a number of factors, including the potential inkilovolts (kV) applied to the X-ray generator, as well as thecomposition of the anode of the generator. For clarity in the presentdisclosure, the values of Hounsfield units for a particular substance orspecies are assumed to be as given in Table I below.

Species/Substance Hounsfield Unit Air −1000 Soft Tissue −300 to −100 Fat−50 Water 0 Blood +30 to +45 Dense Bone +3000

However the numerical values of HUs for particular species (other thanair and water) as given in Table I are to be understood as being purelyillustrative, and those having ordinary skill in the art will be able tomodify these illustrative values, without undue experimentation,according to the species and the X-ray machine used to generate the CTimages referred to herein.

Typically, a translation between HU values and gray scale values isencoded into a DICOM (Digital Imaging and Communications in Medicine)file that is the CT scan output from a given CT machine. For clarity inthe following description the correlation of HU=−1000 to black, andHU=3000 to white, and correlations of intermediate HU values tocorresponding intermediate gray levels is used, but it will beunderstood that this correlation is purely arbitrary. For example, thecorrelation may be “reversed,” i.e., HU=−1000 may be assigned to white,HU=3000 assigned to black, and intermediate HU values assigned tocorresponding intermediate gray levels. Thus, those having ordinaryskill in the art will be able to adapt the description herein toaccommodate other correlations between Hounsfield units and gray levels,and all such correlations are assumed to be comprised within the scopeof the present invention.

In a marking step (block 104) the physician 54 (FIG. 1) marks anintended start point, where he/she will insert medical instrument 21(FIG. 1) into the patient 22 (FIG. 1), and an intended terminationpoint, where the distal end of the medical instrument 21 is toterminate. The two points may be on the same 2D slice. Alternatively,each point may be on a different slice. Typically, but not necessarily,both points are in air, i.e., where HU=−1000, and the termination pointis usually, but not necessarily, at a junction of air with liquid ortissue shown in the slice. An example where the termination point is notat such a junction is when the point may be in the middle of anair-filled chamber.

FIG. 5 illustrates a start point 150 and a termination point 152 thatare marked on the same 2D slice by the physician, and for clarity thesepoints are assumed, except where otherwise stated, to be the points usedin the remaining description of the flowchart. Typically, the start andtermination points are displayed in a non-gray scale color, for example,red.

In a permissible path definition step (block 106), the physician definesranges of Hounsfield units which the path finding algorithm, referred tobelow, uses as acceptable voxel values in finding a path from startpoint 150 to termination point 152. The defined range typically includesHUs equal to −1000, corresponding to air or a void in the path; thedefined range may also include HUs greater than −1000, for example, therange may be defined as given by expression (1):{HU|−1000≤HU≤U}  (1)

where U is a value selected by the physician.

For example, U may be set to +45, so that the path taken may includewater, fat, blood, soft tissue as well as air or a void. In someembodiments, the range may be set by the processor 40 (FIG. 1) withoutintervention of the physician.

There is no requirement that the defined range of values is a continuousrange, and the range may be disjoint, including one or more sub-ranges.In some embodiments a sub-range may be chosen to include a specific typeof material. An example of a disjoint range is given by expression (2):{HU|HU=−1000 or A≤HU≤B}  (2)

where A, B are values selected by the physician.

For example, A and B may be set to be equal to −300 and −100respectively, so that the path taken may include air or a void and softtissue.

The method of selection for the range of HUs may include any suitablemethod, including, but not being limited to, by number, and/or by nameof material, and/or by gray scale. For example, in the case of selectionby gray scale, physician 54 (FIG. 1) may select one or more regions ofthe CT image, and the HU equivalents of the gray scale values of theselected regions are included in the acceptable range of HUs for voxelsof the path to be determined by the path finding algorithm.

In the case of selection by name, a table of named species may bedisplayed to the physician. The displayed table is typically similar toTable I, but without the column providing values of Hounsfield units.The physician may select one or more named species from the table, inwhich case the HU equivalents of the selected named species are includedin the acceptable range of HUs for voxels of the path to be determinedby the path finding algorithm.

In a path finding step (block 108), processor 40 (FIG. 1) implements apath finding algorithm to find one or more shortest paths, between startpoint 150 and termination point 152, that is to be followed by medicalinstrument 21 (FIG. 1). The algorithm assumes that traversable voxels inthe path include any voxels having HUs in the HU range defined in thestep of block 106, and that voxels having HU values outside this definedrange act as barriers in any path found. While the path findingalgorithm used may be any suitable algorithm that is able to determine ashortest path within a three-dimensional maze, the inventors have foundthat the Flood Fill algorithm, Dijkstra's algorithm, or an extensionsuch as the A* algorithm, give better results in terms of speed ofcomputation and accuracy of determining the shortest path than otheralgorithms such as Floyd's algorithm or variations thereof.

In some embodiments, the path finding step includes taking account ofmechanical properties and dimensions of medical instrument 21 (FIG. 1).For example, in a disclosed embodiment, medical instrument 21 may belimited, when it bends, to a range of possible radii of curvature. Indetermining possible paths to be followed by the medical instrument 21,the processor 40 (FIG. 1) ensures that no portion of the path defines aradius less than this range of radii.

In a further disclosed embodiment, the processor 40 (FIG. 1) takesaccount of mechanical properties of the medical instrument 21 (FIG. 1)that permit different portions of the medical instrument 21 differentranges of radii of curvature. For example, the end of a possible pathmay have a smaller radius of curvature than the possible radii ofcurvature of a proximal part of the medical instrument 21. However, thedistal end of the medical instrument 21 may be more flexible than theproximal part, and may be flexible enough to accommodate the smallerradius of curvature, so that the possible path is acceptable.

In considering the possible radii of curvature of the medical instrument21 (FIG. 1), and the different radii of curvature of possible paths, theprocessor 40 (FIG. 1) takes into account which portions of a path needto be traversed by different portions of the medical instrument 21, andthe radii of curvature achievable by the medical instrument 21, as thedistal end of the medical instrument 21 moves from start point 150 totermination point 152.

In a yet further disclosed embodiment, the processor 40 (FIG. 1) ensuresthat a path diameter D is always larger than a measured diameter d ofmedical instrument 21. The confirmation may be at least partiallyimplemented, for example, by the processor 40 using erosion/dilationalgorithms, as are known in the art, to find voxels within the rangesdefined in the step of block 106.

In an overlay step (block 110), the shortest path found in the step ofblock 108 is overlaid on an image that is displayed on display screen56. FIG. 6 illustrates a shortest path 154, between start point 150 andtermination point 152, that has been overlaid on the image of FIG. 5.Typically path 154 is displayed in a non-gray scale color, which may ormay not be the same color as the start and termination points. In thecase that the step of block 108 finds more than one shortest path, allsuch paths may be overlaid on the image, typically in different non-grayscale colors.

Typically, the path found traverses more than one 2D slice, in whichcase the overlaying may be implemented by incorporating the path foundinto all the 2D slices that are relevant, i.e., through which the pathtraverses. Alternatively, or additionally, an at least partiallytransparent 3D image may be generated from the 2D slices of the scan,and the path found may be overlaid on the 3D image. The at leastpartially transparent 3D image may be formed on a representation of anexternal surface of patient 22, as is described in more detail below.

FIG. 7 is a representation of an external surface 180 of patient 22,according to an embodiment of the present invention. Processor 40(FIG. 1) uses the CT scan data acquired in the step of block 100 togenerate the representation of the external surface, by using the factsthat air has a HU value of −1000 while skin has a HU value significantlydifferent from this. By way of example, representation 180 is assumed tobe formed on a plane parallel to the coronal plane of the patient 22,i.e., parallel to an x-y plane of a frame of reference 184 defined bythe patient 22, the axes of which are also drawn in FIG. 7 and in FIG.8.

FIG. 8 schematically illustrates a boundary plane 190 and a boundingregion 192, according to an embodiment of the present invention. Underdirections from physician 54 (FIG. 1), processor 40 (FIG. 1) optionallydelineates regions of representation 180 which are to be renderedtransparent, and those which are to be left “as is.” In order to performthe delineation, the physician defines boundary plane 190, and boundingregion 192 in the boundary plane, using a bounding perimeter 194 for theregion 192.

For clarity, the following description assumes that the boundary planeis parallel to an x-y plane of frame of reference 184, as is illustratedschematically in FIG. 8, and that it has an equation given by:z=z _(bp)  (3)

As described below, processor 40 uses the boundary plane and thebounding region 192 to determine which elements of surface 180 are to berendered locally transparent, and which elements are not to be sorendered.

Processor 40 determines elements of surface 180 (FIG. 7) having valuesof z≥z_(bp), and that, when projected along the z-axis, lie withinbounding region 192. The processor 40 then renders the elementstransparent so that, consequently, these elements are no longer visiblein surface 180. For example, in FIG. 8 a tip 196 of the nose of patient22 has a value z≥z_(bp), so a broken line 198 in the vicinity of thepatient's nose tip illustrates parts of external surface 180 that are nolonger visible when the image of the surface is presented on displayscreen 56 (FIG. 1).

In consequence of the above-defined elements being rendered transparent,elements of surface 180, having values of z<z_(bp) and that whenprojected along the z-axis lie within bounding region 192 are nowvisible, so are displayed in the image. Prior to the local transparentrendering, the “now visible” elements were not visible since they wereobscured by surface elements. The now visible elements include elementsof shortest path 154, as is illustrated in FIG. 9.

FIG. 9 schematically illustrates surface 180 as displayed on displayscreen 56 (FIG. 1) after the local transparency rendering of theelements of the surface within bounding region 192 (FIG. 8). For claritya broken circle 194A, corresponding to bounding perimeter 194 (FIG. 8)has been overlaid on the image, and frame of reference 184 is also drawnin the figure. Because of the transparent rendering of elements withincircle 194A, an area 200 within the circle now shows internal structure,derived from the CT tomographic data received in the step of block 100,of patient 22 (FIG. 1).

Shortest path 154 has also been drawn in FIG. 9. Because of thetransparent rendering of elements within circle 194A, a portion of thepath is now visible in the image of surface 180, and has been drawn as asolid while line 202. The portion of the path that is invisible, becauseit is hidden by elements of surface 180 that have not been renderedtransparent, is shown as broken white line 204.

It will be appreciated that in the case illustrated in FIGS. 7 and 9,the image shown on the display screen 56 is a view of the patient 22 asviewed along the z axis of the x-y plane.

The description above provides one example of the application of localtransparency to viewing a shortest path derived from tomographic data,the local transparency in this case being formed relative to a planeparallel to the coronal plane of the patient 22. It will be understoodthat because of the three-dimensional nature of the tomographic data,the data may be manipulated so that embodiments of the present inventionmay view the shortest path 154 using local transparency formed relativeto substantially any plane through patient 22, and that may be definedin frame of reference 184.

In forming the local transparency, the dimensions and position of theboundary plane 190 and the bounding region 192 may be varied to enablethe physician 54 (FIG. 1) to also view the shortest path 154, andinternal structures in the vicinity of the path 154.

The physician 54 may vary the direction of the bounding plane 190, forexample to enhance the visibility of particular internal structures.While the bounding plane 190 is typically parallel to the plane of theimage presented on display screen 56, this is not a requirement, so thatif, for example, the physician 54 wants to see more detail of aparticular structure, she/he may rotate the bounding plane 190 so thatit is no longer parallel to the image plane.

In some cases, the range of HU values/gray scales selected in the stepof block 106 includes regions other than air, for example, regions thatcorrespond to soft tissue and/or mucous. The path 154 found in the stepof block 108 may include such regions, and in this case, for medicalinstrument 21 (FIG. 1) to follow the path 154, these regions may have tobe cleared, for example by debriding. In an optional warning step (block112), the physician 54 (FIG. 1) is advised of the existence of regionsof path 154 that are not in air, for example by highlighting a relevantsection of the path 154, and/or by other visual or auditory cues.

While the description above has assumed that the CT scan is an X-rayscan, it will be understood that embodiments of the present inventioncomprise finding a shortest path using MRI (magnetic resonance imaging)tomography images.

Thus, referring back to the flowchart 90, in the case of MRI images,wherein Hounsfield values may not be directly applicable, in step 106the physician 54 (FIG. 1) defines ranges of gray scale values (of theMRI images) which the path finding algorithm uses as acceptable voxelvalues in finding a path from the start point 150 to the terminationpoint 152. In the step of block 108, the path finding algorithm assumesthat traversable voxels in the path include any voxels having grayscales in the gray scale range defined in the step of block 106, andthat voxels having gray scale values outside this defined range act asbarriers in any path found. Other changes to the description above, toaccommodate using MRI images rather than X-ray CT images, will beapparent to those having ordinary skill in the art, and all such changesare to be considered as comprised within the scope of the presentinvention.

Reference is now made to FIGS. 10-12. FIG. 10 is a flowchart 300including steps in a method of computing segments of the path 154 andcomputing locations of virtual cameras 320 along the path 154 for use inthe apparatus 20 of FIG. 1. FIGS. 11 and 12, are schematic diagramsillustrating the steps of the method of the flowchart 300 of FIG. 10.

The processor 40 (FIG. 1) is configured to find (block 302) turningpoints 324 in the 3D path 154 above a threshold turning, and computesegments 322 of the 3D path 154 and respective different locations alongthe 3D path 154 of the respective virtual cameras 320 responsively tothe found turning points 324 as shown in FIG. 11.

Sub-steps of the step of block 302 are now described below.

The processor 40 (FIG. 1) is configured to compute (block 304) thesegments 322 based on an n-dimensional polyline simplification. In someembodiments, the n-dimensional polyline simplification includes theRamer-Douglas-Peucker algorithm, the Visvalingam-Whyatt algorithm, orthe Reumann-Witkam, by way of example. Any suitable algorithm whichsimplifies a n-dimensional polyline to a polyline of less dimensions maybe used. The algorithms typically analyze the path 154 removing smallturning points while leaving larger turning points so that the largerturning points 324 define segments 322 between the turning points 324.The threshold turning values corresponding to turning points which areremoved from the path 154 may be set by configuring the parameters ofthe algorithm being used. For example, the input parameter of theRamer-Douglas-Peucker algorithm may be set to about 0.08.

In other embodiments, the processor 40 is configured to compute thesegments 322 using any suitable algorithm so that the turning pointsbelow the threshold turning value are removed.

The processor 40 (FIG. 1) is configured to position (block 306) thevirtual cameras 320 at, or around, the turning points 324 between thesegments 322 as well as at the start point 150 and optionally at thetermination point 152 of path 154. FIG. 11 shows that the path 154 in apassage 328 has been simplified with one turning point 324 and twosegments 322. Three virtual cameras 320 have been respectively placed atthe start point 150, the turning point 324, and the termination point152 on the path 154. Small turning points (indicated using a dotted lineoval shape 326) on the path 154 are removed by the n-dimensionalpolyline simplification to leave the turning point 324.

The processor 40 is configured to check (block 308) a line of sightbetween two adjacent virtual cameras 320 and position one or morevirtual cameras 320 between the two adjacent virtual cameras 320responsively to the line of sight being blocked. The line of sight maybe checked by examining voxels of the 3D CT image to determine if thereis material blocking the line of sight between the adjacent virtualcameras 320. The type of material which is considered to block or allowthe line of sight may be the same as used when computing the path 154 asdescribed with reference to the step of block 106 of FIG. 3. In someembodiments different criteria may be used. For example, the physician54 may set the material that blocks the line of sight as bone and hardtissue, thereby defining air, liquid and soft tissue as materials whichdo not block line of sight. In some cases, the physician 54 may set thematerial that blocks the line of sight as bone, hard tissue, and softtissue. Alternatively, the physician 54 may set the materials that donot block the line of sight such as air or liquid, instead of specifyingthe materials that block the line of sight. In some embodiments, theprocessor 40 is configured to check that a direct line of sight betweentwo adjacent virtual cameras 320 is not blocked. In other embodiments,the processor 40 may be configured to check that the direct line ofsight between the two adjacent virtual cameras 320 is not blockedincluding checking that a given tolerance around the line of sightbetween the two virtual cameras 320 is not blocked. The given tolerancearound the line of sight may have any suitable value. For example, FIG.11 shows that the line of sight between virtual camera 320-2 and virtualcamera 320-3 along the segment 322-2 is blocked by a portion of tissue330. FIG. 12 shows that another virtual camera 320-4 has been addedbetween virtual cameras 320-2 and 320-3. FIG. 11 also shows thatalthough the direct line of sight between virtual cameras 320-1 and320-2 is not blocked, when taking into account a given tolerance aroundthe line of sight, the line of sight expanded by the given tolerance isblocked by a portion of tissue 332. FIG. 12 shows that another virtualcamera 320-5 has been added between virtual cameras 320-1 and 320-2.Once the additional virtual cameras 320 have been added the processor 40may check the line of sight (or the expanded line of sight) between theadjacent virtual cameras 320 based on the initial virtual cameras 320plus the additional virtual cameras 320.

The processor 40 is optionally configured to position (block 310) one ormore additional virtual cameras 320 in the middle of one or more of thesegments 322 responsively to a distance between the existing virtualcameras 320 exceeding a limit. FIG. 12 shows that virtual cameras 320-6and 320-7 have been added in the segment 322-1 responsively to adistance between the existing virtual cameras 320 exceeding a limit. Thelimit may be any suitable limit, for example, but not limited to, in arange of 1 mm to 20 mm, such as 4 mm. The additional cameras 320 aretypically spaced evenly between the existing virtual cameras 320.

Reference is now made to FIGS. 13-15. FIG. 13 is a flowchart 340including steps in a method of selecting a camera for use in theapparatus 20 of FIG. 1. FIGS. 14 and 15 are schematic diagramsillustrating the steps of the method of the flowchart 340 of FIG. 13.

The processor 40 (FIG. 1) is configured to compute (block 342)respective bisectors 350 (FIG. 14) for respective ones of the virtualcameras 320. In some embodiments, the processor 40 is configured tocompute the respective bisectors 350 as respective planes perpendicularto the 3D path 154 at respective locations of the respective virtualcameras 320 on the 3D path 154. FIG. 14 shows the bisectors 350 for eachof the virtual cameras 320. The bisectors 350 may be computed any timeafter the path 154 has been computed up until the time when the medicalinstrument 21 is in close proximity to the respective virtual camera 320as described in the step of block 344 below.

The processor 40 is configured to find (block 344) the closest virtualcamera 320 (e.g., virtual camera 320-7) to the distal end of the medicalinstrument 21 (FIG. 1) responsively to the tracked coordinates of themedical instrument 21 and the known locations of the virtual cameras320. The processor 40 is configured to find (block 346) on which side ofthe bisector 350 of the closest virtual camera 320 (e.g., virtual camera320-7) the tracked coordinates fall. The processor 40 is configured toselect (block 348) one of the virtual cameras 320 to use for renderingthe virtual endoscopic image based on which side the tracked coordinatesof the medical instrument 21 fall with respect to the bisector 350 ofthe closest virtual camera 320 (e.g., virtual camera 320-7). If thetracked coordinates fall on the side of the bisector 350 closer to thecurrent virtual camera 320 (e.g., virtual camera 320-6), the currentvirtual camera (e.g., virtual camera 320-6) is still used. If thetracked coordinates fall on the side of the bisector 350 further awayfrom the current virtual camera 320 (e.g., virtual camera 320-6), thenext virtual camera 320 (i.e. the closest virtual camera 320 (e.g.,virtual camera 320-7)) along the path is selected as the new virtualcamera 320. The steps of blocks 344-346 are repeated intermittently.

The steps of blocks 344-348 are illustrated in FIG. 15. FIG. 15 showstwo virtual cameras 320-6 and 320-7. The virtual camera 320-6 is thecurrent virtual camera being used to render and display the virtualendoscopic image of the passage 328. The virtual camera 320-7 is furtherdown the path 154 with respect to the direction of travel of the medicalinstrument 21 than virtual camera 320-6. In other words, virtual camera320-7 is closer to the termination point 152 (FIG. 11) than virtualcamera 320-6. FIG. 15 shows various possible example positions 352 ofthe distal end of medical instrument 21. All the positions 352 arecloser to the virtual camera 320-7 than to the virtual camera 320-6.Therefore, for all the positions 352 shown in FIG. 15, the virtualcamera 320-7 will be found to be the closest virtual camera 320 in thestep of block 344. Once the closest virtual camera 320-7 is found, theposition of the distal end of the medical instrument 21 with respect tothe bisector 350 of the closest virtual camera 320-7 is examined in thestep of block 346. In the example of FIG. 15, the positions 352-6 arelocated on the side (indicated by arrow 354) of the bisector 350 closestto the virtual camera 320-6 (the current virtual camera) and thepositions 352-7 are on the other side (indicated by arrow 356) of thebisector 350 further away from the current virtual camera 320-6. In theexample of FIG. 15, according to the step of block 348, if the trackedcoordinates of the distal end of the medical instrument 21 are at any ofthe positions 352-6 or similar positions, the current virtual camera320-6 will remain as the selected virtual camera. If the trackedcoordinates of the distal end of the medical instrument 21 are at any ofthe positions 352-7 or similar positions, the virtual camera 320-7 willbe selected as the new virtual camera.

Therefore, the processor 40 is configured to select the respectivevirtual cameras 320 for rendering respective virtual endoscopic imagesresponsively to which side the tracked coordinates (positions 352) ofthe medical instrument 21 (FIG. 1) fall with respect to respectivebisectors 350 of respective virtual cameras 320 closest to the trackedcoordinates.

In other embodiments, the passages may be divided into regions based onthe segments 322 with the virtual cameras 320 being selected accordingto the region in which the tracked coordinates are disposed.

Reference is now made to FIG. 16, which is a flowchart 360 includingsteps in a method of computing an orientation and shifting the locationof one virtual camera 320 for use in the apparatus 20 of FIG. 1.Reference is also made to FIGS. 17 and 18, which are schematic diagramsillustrating the steps of the method of the flowchart 360 of FIG. 16.

The orientation of the optical axis of each virtual camera 320 iscomputed. The orientation may be computed any time after the locationsof the virtual cameras 320 have been computed. In some embodiments, theorientations of the virtual cameras 320 may be computed as each virtualcamera 320 is selected for use as the medical instrument 21 (FIG. 1) isbeing moved along the path 154. One method to computed the orientationof one virtual camera 320 is now described below.

The processor 40 (FIG. 1) is configured to select (block 362) locations370 on the path 154 from one virtual camera 320-2 to another virtualcamera 320-4 as shown in FIG. 17. The locations 370 may be selected tobe inclusive or exclusive of the location of the virtual camera 320-4.The locations 370 may be selected by dividing the segment 322 betweenthe virtual cameras 320-2 and 320-4 into sub-segments. Alternatively,the locations 370 may be selected by measuring a given distance alongthe path 154 from the virtual camera 320-2 to each location 370. In someembodiments, the locations 370 may be selected using points that definethe path 154 when the path 154 was generated. The processor 40 (FIG. 1)is configured to define vectors 372 from the virtual camera 320-2 to thelocations 370 and compute (block 364) an average direction 374 of thevectors 372 as shown in FIG. 17. Therefore, the processor 40 isconfigured to compute an average direction of the vectors from thelocation of the virtual camera 320-2 to different points (e.g., thelocations 370) along the 3D path 154. The processor 40 (FIG. 1) isconfigured to compute (block 366) the orientation of the virtual camera320-2 responsively to the computed average direction 374. In otherwords, the orientation of the optical axis of the virtual cameras 320 iscomputed as the average direction 374. In other embodiments, theorientation may be computed as a direction parallel to the path at thelocation of the respective virtual camera 320-2.

In some embodiments, the respective locations of the respective virtualcameras 320 (e.g., virtual camera 320-2) may be shifted back, forexample, in an opposite direction 376 to the respective averagedirections 374 of the respective virtual cameras 320 as shown in FIG.18. Shifting the virtual cameras 320 back may result in a better view ofthe medical instrument 21 within the respective virtual endoscopicimages particularly when the medical instrument 21 is very close to therespective virtual cameras 320, and may result in a better view of thesurrounding anatomy. Therefore, the processor 40 (FIG. 1) is configuredto shift (block 368) the location of the virtual camera 320-2 in theopposite direction 376 to the computed average direction 374 to a newlocation 380 as shown in FIG. 18. The extent of the shift may be fixed,e.g., a predetermined number of millimeters, such as in a range between0.2 and 2 mm, e.g. 1.3 mm. Alternatively, the extent of the shift may belimited by a surrounding anatomy 378 so that the camera 320-2 is shiftedback as much as possible as long as it is not pushed into bone ortissue, as defined by the physician 54. The type of material considered“surrounding anatomy” may be the same or different criteria as used todefine the material which blocks a path of the medical instrument 21 asdescribed with reference to the step of block 106 of FIG. 3.

The field of view of the respective virtual cameras 320 may be set toany suitable respective values. The field of view of each virtual camera320 may be fixed, for example, in a range of values between 25-170degrees, e.g., 90 degrees. The field of view of any one virtual camera320 may be set according to the outer limits of the segment 322 of thepath 254 that that virtual camera 320 covers (for example, derived fromthe vectors 372 of FIG. 17 or by finding the outer limits of thesegments 322) with an additional tolerance (such as a given angulartolerance for example, by adding X degrees on to the outer limits, whereX may be any suitable value, for example, in a range between 5 and 90degrees) for that virtual camera 320. In some embodiments, the field ofview may be set to encompass all the anatomy in the segment 322 untilthe next virtual camera by analyzing the surrounding anatomy 378 aroundthe segment 322. The type of material considered “surrounding anatomy”may be the same or different criteria as used to define the materialwhich blocks a path of the medical instrument 21 as described withreference to the step of block 106 of FIG. 3.

Reference is now made to FIG. 19, which is a flowchart 390 includingsteps in a method of rendering a transition for use in the apparatus 20of FIG. 1. Reference is also made to FIG. 20, which is a schematicdiagram illustrating the steps of the method of the flowchart 390 ofFIG. 19.

The transition between two virtual cameras 320 and therefore thetransition between the associated virtual endoscopic images may be asmooth transition or a sharp transition. In some embodiments, a smoothtransition between the two respective virtual cameras may be performedby performing the following steps. The transition is described by way ofexample between virtual camera 320-2 and virtual camera 320-4.

The processor 40 (FIG. 1) is configured to find (block 392) locations ofadditional virtual cameras 396 to be added between the current virtualcamera 320-2 and the next virtual camera 320-4 as shown in FIG. 20. Thelocations of the additional virtual cameras 396 may be found in asimilar manner to the selection of the locations 370 described withreference to FIGS. 16 and 17. Any suitable number of additional virtualcameras 396 may be selected. A larger number of additional virtualcameras 396 will generally lead to a smoother transition.

The processor 40 (FIG. 1) is configured to render and display (block394) on the display screen 56 (FIG. 1) a transition between tworespective virtual endoscopic images of two respective adjacent virtualcameras 320-2, 320-4 based on successively rendering respectivetransitional virtual endoscope images of the passage 328 (FIG. 20) inthe body viewed from respective locations of respective additionalvirtual cameras 396 disposed between the two adjacent virtual cameras320-2, 320-4. Each of the transitional virtual endoscope images may bedisplayed on the display screen 56 for any suitable duration, forexample, a duration in the range of 20 to 40 milliseconds.

Reference is now made to FIGS. 21-23, which are schematic virtualendoscopic images 398 rendered and displayed by the apparatus 20 ofFIG. 1. As previously mentioned with reference to FIG. 2, the processor40 (FIG. 1) is configured to render and display on the display screen 56(FIG. 1) the respective virtual endoscopic images 398, based on the 3DCT image, of the passage 328 in the body viewed from the respectivelocations and orientations of the respective virtual cameras 320 (FIG.11) including an animated representation 400 of the medical instrument21 (FIG. 1) positioned in the respective virtual endoscopic images 398in accordance with the tracked coordinates of the medical instrument 21.

FIGS. 21-22 show the virtual endoscopic image 398-1 viewed from thelocation of one virtual camera 320. The representation 400 of themedical instrument 21 is shown in FIG. 21 at one position in the passage328 along the path 154 and in FIG. 22 at a more advanced position in thepassage 328 along the path 154. The path 154 which still remains to betraversed is shown using arrows which disappear as the medicalinstrument 21 moves along the path 154. Any suitably line or symbol maybe used to represent the path 154. FIG. 23 shows the virtual endoscopicimage 398-2 viewed from the location of another virtual camera 320.

The virtual endoscopic images 398 may be rendered using volumevisualization techniques generating the virtual endoscopic images 398from tissue image data (e.g., based on the HU of voxels of the 3D CTscan) in a 3D viewing volume such as a cone projecting outward from thelocation of the relevant virtual camera 320. The images 398 may berendered based on known colors of the tissue. Certain materials such asliquids or even soft tissue may be selected to be transparent while allother denser material may be rendered according to the natural color ofthe denser material. Alternatively, even liquids and soft tissue alongwith the denser material may be rendered according to the natural colorof the respective materials. In some embodiments, the physician 54(FIG. 1) may be set the rendering parameters of the virtual endoscopicimages 398 discussed above. In some embodiments, the surrounding anatomy378 may be rendered whereas other anatomy may be ignored. The type ofmaterial considered “surrounding anatomy” may be the same or differentcriteria as used to define material through which the path 154 cannotpass as described with reference to the step of block 106 of FIG. 3.

The above images 398 are presented only for purposes of illustration,and other sorts of images may likewise be rendered and displayed inaccordance with the principles of the present invention.

Reference is now made to FIGS. 24 and 25. FIG. 24 is a flowchart 500including steps in a method of two-dimensional path visualization foruse in the apparatus 20 of FIG. 1 in accordance with an embodiment ofthe present invention. FIG. 25 is a schematic view of a combined 2D and3D path visualization 512 illustrating the steps of the method of theflowchart 500 of FIG. 24.

The position tracking system 23 (FIG. 1) is configured to track (block502) coordinates of the medical instrument 21 (FIG. 1) within the body.The processor 40 (FIG. 1) is configured to register (block 504) theposition tracking system 23 and a three-dimensional (3D) computerizedtomography (CT) image of at least a part of the body within a commonframe of reference. The step of block 504 is substantially the same asthe step of block 74 described in detail with reference to FIG. 2.

The processor 40 (FIG. 1) is configured to find (block 506) the 3D path154 (FIG. 25) of the medical instrument 21 (FIG. 1) through the passage328 (FIG. 25) passage from the given start point 150 (FIG. 9) to thegiven termination point 152 (FIG. 9) within the common frame ofreference. The step of block 506 is substantially the same as the stepof block 76 described in detail with reference to FIG. 2.

The processor 40 (FIG. 1) is configured to compute (block 508) adirection, and optionally a distance, and/or a 3D vector, to the 3D path154 from the distal end of the medical instrument 21 (FIG. 1)responsively to the tracked coordinates of the medical instrument 21. Insome embodiments, the processor 40 is configured to compute a direction,and optionally a distance, and/or a 3D vector, to a closest point of the3D path 154 from the distal end of the medical instrument 21responsively to the tracked coordinates of the medical instrument 21.

The processor 40 (FIG. 1) is configured to render and simultaneouslydisplay (block 510) on the display screen 56 (FIG. 1) respectivetwo-dimensional (2D) CT slices 514, based on the 3D CT image, includingrespective 2D indications 516 of the direction, and optionally thedistance, and/or the 3D vector, to the 3D path 154 from the distal endof the medical instrument 21 projected onto the respective 2D CT slices514, and optionally a representation 518 of the medical instrument 21responsively to the tracked coordinates of the medical instrument 21.

The 2D CT slices 514 are generated from the 3D CT image according to thetracked coordinates of the distal end of the medical instrument 21 suchthat the 2D CT slices 514 are planes that intersect in the 3D CT imageat the tracked coordinates. In some embodiments, the processor 40 isconfigured to render and simultaneously display, on the display screen56 (FIG. 1), three respective two-dimensional (2D) CT slices 514including respective 2D indications 516 of the direction, and optionallythe distance, and/or the 3D vector, to the (closest point of the) 3Dpath 154 from the distal end of the medical instrument 21 projected ontothe three respective 2D CT slices 514, and optionally a representation518 of the medical instrument 21 responsively to the tracked coordinatesof the medical instrument 21. In some embodiments, the three respective2D slices respectively include a Coronal view slice 514-1, a Sagittalview slice 514-2, and a Transversal view slice 514-3.

The 2D indications 516 may include any suitable symbol such as an arrowor pointer. The 2D indications 516 may indicate only the direction tothe path 154 and therefore all the 2D indications 516 may be the samelength irrespective of the distance to the path 154. In someembodiments, the 2D indications 516 may indicate the direction anddistance (or the 3D vector) to the path 154 and the 2D indications 516may be sized (length and/or width) according to the distance to the path154. The 2D indications 516 represent the direction, and optionallydistance or vector to the path projected on to the respective 2D CTslices 514. For example, based on the frame of reference 184 shown inFIG. 8, the 3D vector is projected on to the x-y plane to generate the2D indication 516 for the Coronal view slice 514-1, on to the y-z planeto generate the 2D indication 516 for the Sagittal view slice 514-2, andon to the x-z plane to generate the 2D indication 516 for theTransversal view slice 514-3.

The representation 518 of the medical instrument 21 (FIG. 1) may includeany suitable symbol, for example, a cross or a circle.

In some embodiments, the processor 40 (FIG. 1) is configured to renderand simultaneously display on the display screen: (a) the respective 2DCT slices 514 including the respective 2D indications 516 of thedirection to the 3D path 154 from the medical instrument 21 projectedonto the respective 2D CT slices 514; and a virtual endoscopic image520, based on the 3D CT image, of the passage 328 in the body viewedfrom at least one respective location of at least one respective virtualcamera including an animated representation 522 of the medicalinstrument 21 positioned in the virtual endoscopic image 520 inaccordance with the tracked coordinates, and a 3D indication 524 of thedirection to the 3D path 154 from the medical instrument 21. The virtualendoscopic image 520 may be rendered and displayed according to themethods described above with reference to FIGS. 2-23.

The above images 514, 520 are presented only for purposes ofillustration, and other sorts of images may likewise be rendered anddisplayed in accordance with the principles of the present invention.

As used herein, the terms “about” or “approximately” for any numericalvalues or ranges indicate a suitable dimensional tolerance that allowsthe part or collection of components to function for its intendedpurpose as described herein. More specifically, “about” or“approximately” may refer to the range of values ±20% of the recitedvalue, e.g. “about 90%” may refer to the range of values from 72% to108%.

Various features of the invention which are, for clarity, described inthe contexts of separate embodiments may also be provided in combinationin a single embodiment. Conversely, various features of the inventionwhich are, for brevity, described in the context of a single embodimentmay also be provided separately or in any suitable sub-combination.

The embodiments described above are cited by way of example, and thepresent invention is not limited by what has been particularly shown anddescribed hereinabove. Rather the scope of the invention includes bothcombinations and subcombinations of the various features describedhereinabove, as well as variations and modifications thereof which wouldoccur to persons skilled in the art upon reading the foregoingdescription and which are not disclosed in the prior art.

What is claimed is:
 1. A medical apparatus, comprising: (a) a medicalinstrument, which is configured to move within a passage in a body of apatient; (b) a position tracking system, which is configured to trackcoordinates of the medical instrument within the body; (c) a displayscreen; and (d) a processor, which is configured to: (i) obtain, basedon a three dimensional (3D) computerized tomography (CT) image, 3D voxeldata representative of at least a part of the body within a common frameof reference, (ii) receive a start point and a termination pointassociated with the passage, (iii) calculate, based on the 3D voxeldata, a 3D path of the medical instrument through the passage from agiven the start point to the termination point, (iv) identify, based onthe 3D voxel data, one or more turning points along the 3D path thatexceed a curvature threshold, (v) provide for user selection a virtualcamera view for each of the one or more turning points via the displayscreen, (vi) identify, based on the tracked coordinates, a desireddirection of the medical instrument, and (vii) display, on the displayscreen, the user selected virtual camera view comprising a virtualrepresentation of the medical instrument relative to two dimensional(2D) CT slices, including respective 2D indications of a desireddirection of the medical instrument.
 2. The apparatus according to claim1, wherein the respective 2D indications of the desired direction to the3D path from the medical instrument include respective arrows.
 3. Theapparatus according to claim 1, wherein the calculating of the 3D pathof the medical instrument through the passage further comprisescalculating a Hounsfield Unit (HU) value for a set of voxels from the 3Dvoxel data.
 4. The apparatus according to claim 1, wherein the processoris configured to identify a point along the 3D path that is closest tothe tracked coordinates of the medical instrument.
 5. The apparatusaccording to claim 4, wherein the processor is configured to compute a3D vector relative to the point and the tracked coordinates of themedical instrument.
 6. The apparatus according to claim 5, wherein the2D indications are based on the 3D vector.
 7. The apparatus according toclaim 1, wherein the virtual camera view further comprises threerespective two-dimensional (2D) CT slices including three respective 2Dindications of the direction.
 8. The apparatus according to claim 7,wherein the three respective 2D slices respectively include a Coronalview, a Sagittal view, and a Transversal view.
 9. The apparatusaccording to claim 1, wherein the processor is further configured toidentify, based on the 3D voxel data, one or more non-air sections alongthe 3D path; wherein the virtual camera view further comprises at leastone visual cue associated with the non-air sections.
 10. The apparatusaccording to claim 1, wherein virtual camera view further comprises avirtual endoscopic image, based on the 3D voxel data, of the passage inthe body viewed from at least one respective location of at least onerespective virtual camera including an animated representation of themedical instrument positioned in the virtual endoscopic image inaccordance with the tracked coordinates, and a 3D indication of thedirection to the 3D path from the medical instrument.
 11. The apparatusaccording to claim 1, wherein the position tracking system comprises anelectromagnetic tracking system, which comprises one or more magneticfield generators positioned around the part of the body and a magneticfield sensor at a distal end of the medical instrument.
 12. A medicalmethod, comprising: (a) tracking coordinates of a medical instrumentwithin a body of a patient using a position tracking system, the medicalinstrument being configured to move within a passage in the body of thepatient (b) obtaining, based on a three dimensional (3D) computerizedtomography (CT) image, 3D voxel data representative of at least a partof the body within a common frame of reference; (c) receiving a startpoint and a termination point associated with the passage; (d)calculating, based on the 3D voxel data, a 3D path of the medicalinstrument through the passage from the start point to the terminationpoint; (e) identifying, based on the 3D voxel data, one or more turningpoints along the 3D path that exceed a curvature threshold; (f)providing, on a display screen, for user selection a virtual camera viewfor each of the one or more turning points; (g) identifying, based onthe tracked coordinates, a desired direction of the medical instrument;and (h) displaying, on the display screen, the user selected virtualcamera view comprising a virtual representation of the medicalinstrument relative to two-dimensional (2D) CT slices, includingrespective 2D indications of a desired direction of the medicalinstrument.
 13. The method according to claim 12, wherein the respective2D indications of the desired direction to the 3D path from the medicalinstrument include respective arrows.
 14. The method according to claim12, wherein calculating of the 3D path of the medical instrument throughthe passage further comprises calculating a Hounsfield Unit (HU) valuefor a set of voxels from the 3D voxel data.
 15. The method according toclaim 12, wherein the computing includes identify a point along the 3Dpath that is closest to the medical instrument.
 16. The method accordingto claim 15, wherein the computing includes computing a 3D vectorrelative to the point and the tracked coordinates of the medicalinstrument.
 17. The method according to claim 16, wherein the 2Dindications are based on the 3D vector.
 18. The method according toclaim 12, wherein the virtual camera view further comprises threerespective two-dimensional (2D) CT slices including three respective 2Dindications of the direction.
 19. The method according to claim 18,wherein the three respective 2D slices respectively include a Coronalview, a Sagittal view, and a Transversal view.
 20. The method accordingto claim 12, further comprising identifying, based on the 3D voxel data,one or more non-air sections along the 3D path, wherein the virtualcamera view further comprises at least one visual cue associated withthe non-air sections.
 21. The method according to claim 12, wherein thevirtual camera view further comprises a virtual endoscopic image, basedon the 3D voxel data, of the passage in the body viewed from at leastone respective location of at least one respective virtual cameraincluding an animated representation of the medical instrumentpositioned in the virtual endoscopic image in accordance with thetracked coordinates, and a 3D indication of the direction to the 3D pathfrom the medical instrument.
 22. A software product, comprising anon-transient computer readable medium in which program instructions arestored, which instructions, when read by a central processing unit(CPU), cause the CPU to: (i) track coordinates of a medical instrumentwithin a body of a patient using a position tracking system, the medicalinstrument being configured to move within a passage in the body of thepatient; (ii) obtain, based on a three dimensional (3D) computerizedtomography (CT) image, 3D voxel data representative of at least a partof the body within a common frame of reference; (iii) receive a startpoint and a termination point associated with the passage; calculate,based on the 3D voxel data, a 3D path of the medical instrument throughthe passage from the start point to the termination point; (iv)identify, based on the 3D voxel data, one or more turning points alongthe 3D path that exceed a curvature threshold; (v) provide for userselection a virtual camera view for each of the one or more turningpoints via a display screen; and (vi) display, on the display screen theuser selected virtual camera view comprising a virtual representation ofthe medical instrument relative to two-dimensional (2D) CT slices,including respective 2D indications of a desired direction of themedical instrument.